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ABSTRACT 

The National Assessment of Educational Progress 
(NAEP) serves as the nation's primary indicator of what school 
children know and can do. This study examines the policy relevant 
variable, computer use in the classroom, as indicated by the 1992 
NAEP Trial State Assessment data for Mississippi. After a preliminary 
examination of other variables relevant to educational policy, a 
hierarchical linear model analysis was performed for the composite 
mathematics score within the eighth grade in Mississippi. Gender, 
race-ethnicity, and socioeconomic differences in scores were 
identified, and their correlation with school and student variables 
was explored. In Mississippi, 53% of students did not have computers 
available in their classrooms. While neither avai 1 ab i 1 i ty/access nor 
frequency of computer use was found to be s tat is t i cal ly significantly 
associated with average school achievement, both variables showed 
associations with the gap between minority and nonminority students. 
In schools where computers were the most available and accessible, 
the gap between minority and nonminority students was significantly 
reduced. Implications of these findings for educational policy are 
provocative. (Contains 6 tables, 9 figures, and 18 references.) 
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Abstract 

Recent emphasis has been made on improving education through educational reform. 

The National Assessment of Educational Progress (NAEP) has played an increased role in 
measuring student achievement, especially at the state level. It serves as the Nation's primary 
indicator of what school children know arid can do and has become "The Nation's Report Card". 
NAEP data is a rich source of information awaiting analysis. It has the potential of providing 
tremendous insights to policy makers and educational planners. However, due to its complexity, 
few researchers have attempted to analyze the data for specific policy implication. 

This study will provide an examinatipn of the policy relevant variable, computer use in 
the classroom, contained in the 1992 National Assessment of Educational Progress Trial State 
Assessment (NAEP-TSA) data for Mississippi. It will address the potential relationship between 
computers in the classroom and student mathematics performance. 

Computers can be used in a wide variety of ways in mathematics classrooms. Although 
they may be most frequently used for computational drill and practice, teachers can take full 
advantage of this technology by using computers to teach graphs, spreadsheets, and extended 
investigations of mathematical ideas. The computer has the potential to provide opportunities 
.for problem solving using "hands-on" techniques and also can be effective as a tool in small 
group. This study will use information from students and teachers included in the NAEP-TSA 
sample to construct variables to assess computer use and availability and measure the 
relationship between these variables and mathematics performance. 

Educational researchers have been studying "school effects" for quite sometime. In such 
studies the researcher seeks to identify characteristics in schools that are associated with student 
outcomes, separate from the characteristics of the students themselves. Thus, the purpose is to 
understand why some schools seem to be better able to produce positive educational outcomes 
than other schools. A major concern in studying school effects has been to take into account and 
measure accurately the factors at all levels that could influence achievement. 

Following a preliminary examination of policy-relevant variables, a hierarchical linear 
model (HLM) analysis will be performed for the composite mathematics score within eighth 
grade in Mississippi. In addition, gender, race-ethnicity, and socioeconomic status differences in 
these scores will be identified and the their correlation with school and student variables will be 
explored. 

The variables from the NAEP/TSA data to be used in the analysis are: (1) Student-level 
Variables: gender, race/ethnicity or minority/non-minority, socioeconomic status; (2) School- 
level Variables (Instructional characteristics of the school): computer use in the classroom and 
availability/access of computers. 

The results of this study will address a very important policy question about the 
association between the use and availability of computers in the classroom and student 
proficiency. The conclusions can readily impact institutional policy and instructional changes at 
the local and state policy levels. The secondary analysis of NAEP data, along with the use of 
HLM, holds promise of providing educational practitioners and policy makers with ric&». 
information about possible school and classroom influences. 




3 



NAEP-TSA and Computers 3 



Introduction 

This paper is a part of a project funded by a grant received from the U S. Department of 
Education, National Center for Education Statistics (R999850010). This paper reports on only 
one of the components funded by the grant. It will analyze and report on the data from the 1992 
National Assessment of Educational Progress Trial State Assessment (NAEP-TSA) in 
Mississippi. The primary purpose of this study is to develop a model for analyzing state data that 
researchers in state agencies will fmd useful. It uses state-of-the-art hierarchical linear modeling 
(HLM) statistical procedures to investigate policy-relevant instructional variables that impact on 
students' mathematics performance. The results can be used to provide educational policy 
makers and planners with information for the improvement of educational programs and 
practices. 

With national emphasis on improving education, NAEP has played an increased role in 
measuring student achievement. With the assistance of NAEP and the "National Report Card,” 
Americans may identify states in need of improvement. However, just a ranking of states is not 
very useful. What is needed is useful information to impact policy. Most of the states in the 
nation have implemented State Report Cards as a measure of accountability and student 
achievement to provide the public and policy makers with comparable information on the 
progress of state education programs. However, states do not make adequate use of NAEP data. 
While NAEP data is a rich source of information awaiting analysis, few researchers have 
attempted to analyze the data for specific policy implication. Since the National Educational 
Goals Panel is establishing ways to report NAEP data, this project will provide a modeHor using 
NAEP-TSA analyses on student achievement as a part of state report cards. 
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A review of the results from the 1992 NAEP data confirms what many already believed, 
that students in the southeastern region of the U. S. are substantially behind the rest of the nation 
in mathematics achievement. Nationally, students in grade four had an average mathematics 
proficiency of 217 while students in the southeast had an average of 209. A similar pattern is 
seen for students in grade eight, 268 nationally and 258 for the southeast. Further, two percent of 
the nations' fourth grade students and three percent of the eighth grade students performed above 
average in mathematics achievement while only one percent performed above average in the 
southeast. The nation as a whole had 39 percent of the eighth grade students performing below 
basic proficiency while the southeastern region had 47 percent. In the fourth grade the gap is 
even wider - 41 percent for the nation and 52 percent for the southeast. With such a deficit, this 
team of researchers challenged each other to develop a model for using NAEP-TSA data to 
identify policy-relevant variables that could lead to improving mathematics achievement in the 
southeastern states. 

In short, students in the southeastern region of the U.S. are performing below the nation 
as a whole and students in Louisiana, Mississippi, and North Carolina are scoring among the 
lowest in the southeast. As educational planners and policy makers continue to strive to improve 
education in these states, information from the NAEP data should be utilized. Information from 
the NAEP-TSA data can provide information for decision makers when considering such 
questions as: 

1 . Should education policy makers expect to see mathematics achievement improve 
if they significantly increase technology funds for computers in classrooms? 

2. Would scarce educational funds be better spent helping teachers understand new 
methods of delivering mathematics instruction? 
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3. Should states and local school districts develop policies that would encourage 
students and parents to spend more time on homework? 

NAEP-TSA provides us with the data and HLM statistical techniques provide us with the 
method to examine policy-relevant activities and practices. While factors appear to be related to 
eighth grade public school students' proficiency in mathematics, cause-and-effect links between 
various contextual factors and students' mathematics proficiency may not be established. 
However, this paper is designed to reveal promising policy implications about the relationship 
between computer use and/or availability, various student body characteristics, and mathematics 
performance. 

Using NAEP Data for Studying Instructional Policy 

As educational planners and policy makers continue to strive to improve education, this 
team of researchers believes that information from the NAEP-TSA data should be utilized. This 
paper focuses on a set of policy questions that address, "how students' mathematics performance 
is influenced by the availability and use of computers.” This relationship has implications for 
policy makers at three different levels: (a) classroom, (b) school, (c) and state. 

Educational researchers have been studying "school effects" for a long time. (See 
Brookover, Beady, Flood, Schweitzer, and Wisenbaker (1979), Rutter (1983), Good and Brophy 
(1986), Good and Weinstein (1986), Weinstein (1986), Oakes (1989), Arnold, Kaufman and 
Sedlacek (1992), and Sedlacek (1991). In such studies the researcher seeks to identify 
characteristics in schools that are associated with student outcomes, separate from the 
characteristics of the students themselves. Thus, the purpose is to understand why some schools 

-m. 

seem to be better able to produce positive educational outcomes than other schools. A major 
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concern in studying school effects has been to take into account and measure accurately the 
factors at all levels that could influence achievement (Burstein, 1980). 

The practices and policies at the local school level have a direct influence on student 
achievement. Local schools influence instruction through such policies as student access to 
computers by providing computer laboratories, classroom computers, computer take-home 
policies, etc. Local school boards make policies that impact mathematics instruction and student 
achievement when they make resource allocation decisions that effect instruction such as 
student/computer ratio and student access to computers. State policy makers can also have a 
positive impact on school practices through programs that encourage and provide adequate 
access to computer usage, instructional methodology, and instructional practices. This study will 
provide information from which policy recommendations can be made to improve mathematics 
performance. 



Theoretical Framework 

This project takes its lead from the landmark action of the National Council of Teachers 
of Mathematics (NCTM) which set standards for mathematics curriculum and teaching. 
According to NCTM, all students must learn more mathematics, and sometimes, different 
mathematics to improve this mathematics proficiency. Mathematics instruction has been 
characterized by extensive use of textbooks and worksheets (Romberg and Carpenter, 1980). 
What students learn depends, to a great degree, on how the instruction has been presented to 
them. According to NCTM, classroom instruction needs to be more student centered (NCTM, 
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NCTM recommends well-equipped classrooms and instruction reflecting the vitality of 
mathematics. They also recommended the use of small groups and cooperative-learning 
strategies for mathematics teaching. Educators argue that teaching must move away from a 
computation-dominated curriculum to a curriculum that stresses problem solving and 
understanding (Taylor, 1990). Small group instruction is appropriate for this type of curriculum. 
Further, this approach mirrors the use of mathematics in the workplace and reduces mathematics 
anxiety (Mullis et al., 1991). 

NCTM standards recommend that schools take a look at how teachers are providing 
mathematics instruction. Results of a survey of elementary teachers indicate that: (a) most 
teachers use few materials other than textbooks in mathematics instruction, (b) use of most 
materials tends to decrease as grade level increases, (c) calculators are rarely used, but usage 
tends to increase with grade level, and (d) most teachers report they want more materials (Scott, 
1993). 

A central purpose of this study is to describe a set of policy-relevant variables associated 
with student mathematics achievement. Specifically, access and use of computers to improve 
mathematics performance of eighth grade students has relevant policy implication for both local 
and state policy makers. 

Purpose of Study 

NAEP has played an increased role in measuring student achievement, especially at the 
state level. It serves as the Nation's primary indicator of what school children know and can do 
and has become “The Nation's Report Card”. NAEP data is a rich source of informatioarawaiting 
detailed analysis. It has the potential of providing tremendous insights to policy makers and 
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educational planners. However, due to its complexity, few researchers have attempted to analyze 
the data for specific state and local policy implications. This study will provide a preliminary 
examination of policy relevant variables of interest contained in the 1992 NAEP-TSA data in 
Mississippi. It addresses the potential relationship between policy-relevant instructional practices 
and student mathematics performance of eighth grade students attending public schools. 

In the NAEP data, the students' mathematics teachers stated that more than 50% of eighth 
graders, nationwide, have never or hardly ever used a computer in mathematics classes. In the 
southeast, this rate was 62%. In Mississippi, 80% of the teachers reported that they never, or 
hardly ever, used a computer. 

Only 1 8% of the students reported using a computer at least once a week and 52 percent 
reported that they never used a computer in class. From the students' perspective, 50 percent of 
the fourth graders, 69% of the eighth graders, and 66% of the twelfth graders taking 
mathematics reported never using a computer in mathematics class. (NCES, 1991) 

For the states participating in the 1992 Trial State Assessment, teachers reported low 
availability of computers in public-school eighth-grade mathematics classrooms. For the nation 
as a whole, twenty-four percent of the eighth graders had teachers who reported that computers 
were not available. In the southeast, twenty-nine percent of the eighth graders had teachers who 
reported not having computers available. For Mississippi, this number jumps to fifty-three 
percent. 

A major goal of this study is to identify policy-relevant variables that have an impact on 
student academic performance. One such variable that policy makers can have influence over is 
how computers are used in education. Computers can be used in a wide variety of ways in 
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mathematics instruction. Computers are frequently used for computational drill and practice, 
however, teachers can take full advantage of this technology by using computers to teach graphs, 
spreadsheets, and extended investigations of mathematical ideas. The computer has the potential 
to provide opportunities for problem solving using "hands-on" techniques and can be an 
effective tool in a small group (Male, 1990). This study uses information from students and 
teachers included in the NAEP-TSA data to construct measures of computer use and availability 
and identify relationships between these variables and mathematics performance. 

Computer Availability/ Access and Use 

Recommendations for improving mathematics education often include more use of 
computers. The NCTM has proclaimed the benefits of computers by stating that (a) A computer 
should be available in every classroom for demonstration purposes, and (b) every student should 
have access to a computer for individual and group work (NCTM, 1991). 

NCTM suggests that computers, along with calculators, provide a basis for more 
complex problem-solving situations that engage students in mathematics learning. Incorporating 
computers as integral parts of classroom instruction can help make school mathematics become 
more like the mathematics people actually use in their everyday lives and on the job. The 
availability of increasingly sophisticated and economically feasible technology, such as 
computers, can be used to accelerate the pace of student learning in mathematics (NCTM, 1991). 

Included in the NAEP-TSA data are responses from a questionnaire given to both school 
administrators and teachers about the availability of computers in mathematics classrooms. 
A dmini strators were asked if computers were available all the time in mathematics classrooms; 
if computers were grouped in a laboratory available to mathematics classes; and if computers 
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were available to bring to the classroom when needed. Teachers and students were asked about 



the frequency of computer use. Teachers were asked about how often their students use 
computers in their classes. Students were asked, “In mathematics class, how often do you do the 
following? - Use a computer at least once a week; less than once a week; never.” Teachers were 
also asked to estimate the time that students spend each week working with computers to solve 
mathematics problems - none; 15 minutes; 30 minutes. 

Policy-Relevant Issues To Be Investigated 

1. Do schools with greater availability/access to computers have higher 
achievement levels? 

2. Do schools where students use computers more frequently have higher 
achievement levels? 

3. Do schools with greater availability/access to computers have a smaller gap 
between minority students and nonminority students? 

4. Do schools where students use a computer more frequently have a smaller gap in 
achievement between minority students and nonminority students? 

5. Do schools with greater availability/access to computers have a smaller gap in 
achievement between female and male students? 

6. Do schools where students use a computer more frequently have a smaller gap in 
achievement between female and male students? 

7. Do schools with greater availability/access to computers have more or less of a 
difference in achievement between high and low SES students? 

8. Do schools where students use a computer more frequently have more or less of a 
difference in achievement between high and low SES students? 



NAEP-TSA and Computers 1 1 

Method 

This study investigates data in the NAEP-TSA to determine if the data would support a 
relatively new statistical method called “hierarchical linear modeling” (HLM) (Bryk and 
Raudenbush, 1992). This method of analysis offers several advantages. First, one can explain 
student achievement as a function of school level characteristics while taking into account the 
variance of student outcomes within the school. Second, one can model the effects of student 
characteristics, such as gender, race-ethnicity, and socioeconomic status, on achievement within 
schools. This allows for the explanation of differences in the effects between schools serving 
students with different characteristics. A third advantage is that one can model the between and 
within-school variance at the same time, thus producing more accurate estimates of student 
outcomes. Finally, one can derive better estimates of the predictors of student outcomes within 
schools and classrooms by “borrowing” information about these relationships from other schools 
and classrooms. 

This paper includes the examination of the policy relevant variables of interest to be used 
in the development of linear models that seek to explain variation in mathematics achievement 
scores, using both classroom level and school level variables as explanatory variables. 
Additionally, instructional characteristics will be obtained by aggregating information from 
teacher data to the school level for grade analysis. HLM allows the prediction of student level 
outcomes as a function of school factors, while controlling for student level factors. Thus, the 
multi-level nature of the data can be modeled. HLM also provides for the identification of 
differences by gender, race-ethnicity and socioeconomic status in the correlation of school level 
factors with achievement. 
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The variables from the NAEP-TSA data to be used in the analysis are: (a) Student-level 
Variables: gender, race/ethnicity or minority/nonminority, socioeconomic status; (b) School- 
level Variables (Control Variables): percentage of student body that is minority/nonminority, 
percentage of females in the student body, and average socioeconomic level of the student body; 
and (c) School-level Variables: computer availability and use. 

Variables Used in the Model 

Student level variables 



The student level variables for the eighth grade sample in the state of Mississippi were 
based on 2,498 original cases. The following level 1 variables were included in this study: 
DRACE derived race of student (recoded into 0 = nonminority, 1 = minority) 
DSEX derived sex of student (0 = male, 1 = female) 

SES composite of 6 variables related to parents' education and 

home environment 

B007501 A frequency of computer use for schoolwork 
School level variables 

The school level variables were based on 93 cases in the state of Mississippi. The 
following school level variables were included in this study: 

MDRACE proportion of minorities in student body (average of DRACE) 

MDSEX proportion of females in student body (average of DSEX) 

MSES average SES level of student body 

MCUSE average frequency of computer use .«•- 



COMP AVAL computer availability/access 



NAEP-TSA and Computers 13 

The computer availability/access variable was created using three school level variables 
and one teacher response. The following three school level variables were chosen because they 
to relate to computer availability in a classroom: 

C029301 Are computers always available in math classrooms? 

C029302 Are computers grouped in a lab for math classes? 

C029303 Are computers available to bring to math classes? 

In order to categorize schools, the following ordered triples of the form (C029301, C029302, 
C029303) were generated with 1 indicating a 'yes' response to the corresponding variable: 

( 0 , 0 , 0 ) ( 1 , 0 , 0 ) ( 0 , 1 , 0 ) ( 1 , 1 , 0 ) 

( 0 , 0 , 1 ) ( 1 , 0 , 1 ) ( 0 , 1 , 1 ) ( 1 , 1 , 1 ) 

The next step was to utilize the teachers’ response to question T04520 1 - Are computers 
available for your class? This variable consisted of the following three responses : (a) not 
available, (b) difficult, and (c) available. The school categories were cross tabulated with the 
teacher response to determine the proportion of teachers in each school category who said that 
computers were either difficult or available. The final computer availability/access variable 
represents the proportion of teachers in each category of school who indicated that computers 



who accessible. See Table 1. 
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Table 1 

Final values of computer availabilitv/access variable 



School Level Computer Availability Final Computer 



C029301 


C029302 


C029303 


Availability/ Access 


0 


0 


0 


26.83 


0 


0 


1 


48.94 


0 


1 


0 


59.65 


1 


0 


0 


71.48 


0 


1 


1 


73.30 


1 


1 


0 


100 


1 


0 


1 


N/A 


1 


1 


1 


N/A 



HLM Specifications 

Let yy, DRACEy, DSEXy, SESy, and CUSEy be the plausible mathematics achievement 
score, derived race, derived sex, socioeconomic status, and computer use of the ith student in the 
jth school, respectively. For the intercept to be interpreted as the average achievement of a given 
school, student level predictors were centered. Now let MDRACEj, MDSEXj, MSESj, and 
MCUSEj be average values of the derived race, derived sex, socioeconomic status, and computer 
use of the jth school. Respectively. Since derived race and derived sex are dichotomous 
variables, MDRACEj and MDSEXj can be viewed as the proportion of minority and proportion 
of females in the sample of students selected from the jth school. Further, MSESj and MCUSEj 
can be used as school level SES and computer use measures. Now, the centered student level 
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predictor variable for the ith student in the jth school can be given as CRACEy = DRACEy - 
MDRACEj, CSEXy = DSEXy - MDSEXj, CSESy = SESy - MSESj, and CCUSEy = CUSEy - 
MCUSEj. 

Subject to the notation introduced above, the following HLM models at the student level 
(LEVEL- 1) and at the school level (LEVEL-2) were hypothesized. 

LEVEL- 1 MODEL 

yy = Poj + PraceJ CRACEy + P sexJ CSEXy + p sraj CSESy + P^cjCUSEy + 6y 

for j = 1, . . k and I = 1, . . nj where k is the number of schools sampled and nj is the 
number of students sampled from school j . 

i 

In the LEVEL- 1 model, p 0 j can be viewed as the average school achievement of the jth 
school. Since the predictor variables have been dichotomized, p race j and P, ex j can be viewed as 
the race and gender gaps for the jth school. In the LEVEL- 1 model, it is assumed that, 
conditionally, ey - i.i.d.N(0,o 2 ). (i.e. Conditioned on the random beta parameters, achievement 
scores within a school are assumed to be independently, identically, and normally distributed 
with mean zero and variance a 2 . 

LEVEL-2 MODELS 

At the school level, relationships between the race and gender gaps, and SES and 
computer use slopes and the computer availability measures described earlier were studied. In 
addition, the effects of the proportion of minority, proportion of females, average SES, and 
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average computer use at the school level on the race and gender gaps as well as the SES and 
computer use slopes were of interest. 
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Then, at LEVEL-2, we hypothesized 

pj = rWj + Uj for j = 1, . . ., k for the “gap” and “slope” analyses. 



In the LEVEL-2 models above, it is assumed that Uj ~ i.i.dMVN 5 (0,t) for j = 1, . . ., k. 
That is, the variance covariance matrices are assumed to be equal across schools. (Empirically, 
in our analyses, this assumption does not appear to hold true.) 
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HLMPV Software 

NAEP provides five plausible mathematics achievement scores for each student. Five 
HLM analyses need to be conducted and the results combined according. to the method described 
in the NAEP technical report. The HLMPV software which was used to conduct the analyses 
provided the following results. 

1 . Generalized least squares estimators of regressor parameters in level-2 models 
(i.e. y-parameters), their standard errors, and associated t-test statistics for testing 
Ho ; Yy = 0 versus y^ * 0. 

2. Maximum likelihood estimators of the variance component estimators, T qq ‘s. (i.e. 
qth diagonal elements of t) and the associated chi-square test for testing FU r qq = 

0 versus H,: T qq > 0. 

3. Average reliability measure for each randomly varying regressor parameter in the 
LEVEL- 1 model. The variability of the estimate of a LEVEL- 1 random regressor 
parameter consists a component due to the sampling error and a component due to 
the variability of the parameter itself. Average reliability measures are, in general, 
the proportion of the variability of parameter estimates that can be attributed to 
the parameter variability. 

Cautions When Using HLM 

When analyzing NAEP-TSA data and reporting results using HLM, caution must be 
taken not to abuse or misinterpret the data. To this end, several observations are in order. First, 
in this analysis, the reliabilities associated with the slope parameters were found to be law. This 
could be due to low slope parameter variance estimates indicating relative stability of slope 



NAEP-TS A and Computers 1 8 



parameters across school. Additionally, low reliabilities could be due to an excessive sampling 
error variance estimate. This excessive sampling error variance estimate may be due to under 
specification of the LEVEL- 1 model (i.e inadequate number of explanatory variables in the 
LEVEL- 1 model). If the LEVEL- 1 model is under specified, the resulting slope parameter 
estimates will be biased which may make the LEVEL-2 model parameter estimates, their 
variance estimates, and t-tests inaccurate. 

Second, in the LEVEL- 1 and LEVEL-2 models, linear relationships are assumed. In 
some cases, as seen from the attached graphs, relationships may not be linear. The preferred 
method of avoiding this problem is to dichotomize all of the predictor variables in the LEVEL- 1 
and LEVEL-2 models. However, this must not be done arbitrarily since the manner in which a 
variable is dichotomized may influence the overall results. Further, dichotomizing variables can 
lead to a loss of important information. 

Finally, HLM models assume that the predictor variables are measured without error. 
While gender and race are less susceptible to measurement error, other variables may be 
subjected to a large amount of measurement error since they are self reported by eighth grade 
students. This measurement error not only contributes to the overestimation of the sampling 
error variance but depending to the predictors can also lead to the violation of homogeneity of 
variance assumptions made in the LEVEL- 1 and LEVEL -2 models (e.g. variability of the 
average school achievement of low SES schools and high SES schools may be smaller than the 
variability associated with medium SES schools). 
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Data Source 

Data that was received from the National Center for Educational Statistics included (a) 
NAEP Trial State Assessment Almanacs and (b) data files for the state of Mississippi. This data 
was provided to the researchers under special confidentiality codes. 

Results of HLM Analysis 

Within-school models 

Table 2 shows the results of the unconditional model for the mathematics achievement of 
grade 8 students in Mississippi. The within-school models provide the average within-school 
parameter estimates. The table illustrates that the average achievement score in math for grade 8 
students in Mississippi is 249.79. Because of the centering that was performed on the student 
level predictor variables, the achievement scores represent the average achievement in each 
school at the average gender, race/ethnicity, SES, and frequency of computer use. 



Table 2 

Results from unconditional HLM model 



WITHIN-SCHOOL 

PARAMETER 


Coefficient 


Reliability 


t-ratio 


p-value 


INTERCEPT 


249.79 


0.78 


151.55 


0.001 


RACE/ETHNICITY 


-31.3 


0.16 


-12.83 


0.001 


GENDER 


-4.85 


0.13 


-2.648 


0.008 


SES 


5.74 


0.06 


2.36 


0.018 


COMPUTER USE 


2.01 


0.18 


1.76 


0.078 
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Table 2 also contains the reliability and parameter variance estimates for each of the 
within-school parameters. The reliability is the proportion of the total variance that is parameter 
variance. This parameter variance is what will try to be explained in the between-schools 
models. The reliability estimates contained in Table 2 range from a maximum of .78 for the 
intercept parameter to a minimum of .06 for the SES parameter. These results indicate that 
approximately 78% of the variance around the intercept parameter is parameter variance and has 
the potential to be explained. Similarly, 16%, 13%, and 6% of the variance surrounding the 
race/ethnicity, gender, and SES parameters, respectively, are parameter-variance that will try to 
be explained with the school level variables. 

Between-school models 

Tables 3 through 6 contain the results of the between school models. The first model 
attempts to explain the parameter variance surrounding the within-school intercept. The results 
in Table 3 show that the percentage of minority students in a school has a significant association 
with the average achievement of the school. The negative coefficient indicates that lower 
average school achievement is associated with higher percentages of minority students. This 
result is further illustrated in Figure 1. The results show that no other school level predictor was 
significant at the . 10 level; however, the average frequency of computer use for the school had a 
p-value of .18. 




-* 1 *- 
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Table 3 

Predictors otaverage school achievement for grade 8 



WITHIN SCHOOL PARAMETER 
Between-school predictor 


Coefficient 


T-ratio 


p-value 


INTERCEPT (AVERAGE ACHIEVEMENT) 








Intercept 


271.07 


13.30 


0.0001 


Proportion of minorities 


-32.41 


- 7.12 


0.0001 


Proportion of females 


6.00 


0.52 


0.601 


Average SES 


5.51 


0.50 


0.618 


Computer availability/access 


4.26 


0.62 


0.535 


Average frequency of computer use 


4.61 


1.34 


0.179 


Reliability 




.60 




Parameter variance 




72.00** 




Proportion of parameter variance explained 




.61 





Note: **p s .01 



Table 3 also shows that the reliability was .60 and the parameter variance, although less 
than the conditional model, was still significantly different from 0 indicating that there is more 
parameter variance that can be explained. Finally, the results show that the model was able to 
explain 61% of the original parameter variance. 

The results of the between-schools model for the race/ethnicity parameters are contained 
in Table 4. The results indicate that the average SES level of the students within a school is 
significantly associated with the achievement difference between minority and nonminority 

Ae*. 

students. The direction of the coefficient indicates that higher levels of SES are associated with 
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negative values of the race/ethnicity coefficient which means that the achievement of minority 
students is less than nonminority students. 



Table 4 

Predictors of race/ethnicitv coefficients for grade 8 



WITHIN SCHOOL PARAMETER 
Between-school predictor 


Coefficient 


T-ratio 


p-value 


RACE/ETHNICITY 








Intercept 


-51.10 


-1.06 


0.289 


Proportion of minorities 


-5.05 


-0.40 


0.689 


Proportion of females 


-18.95 


-0.75 


0.451 


Average SES 


-48.73 


-2.01 


0.044 


Computer availability/access 


25.83 


1.90 


0.057 


Average frequency of computer use 


12.49 


1.60 


0.109 


Reliability 




.12 




Parameter variance 




48.66* 




Proportion of parameter variance explained 




.31 





Note: * £ s .05 



Since race/ethnicity is a dichotomized variable, the coefficients associated with this 
variable can be viewed as gaps between the performance of minorities and nonminorities within 
schools. Therefore, the results indicate that as the average SES level of the student body 
increases the coefficient or gap will increase. This is shown in Figure 2. Because the intercept 
term is negative, a decreasing coefficient is moving away from zero; therefore, the gap is 
increasing 
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The results of the between-schools model for the race/ethnicity parameter also indicates 
that the availability/access of computers has a significant association with the achievement 
difference between minorities and nonminorities. The positive coefficient associated with this 
variable indicates that higher levels of availability/access of computers are associated with 
higher race/ethnicity coefficients. Figure 3 illustrates this result and shows that because the 
intercept is negative, increasing the value of the coefficient is moving the coefficient closer to 
zero and therefore reducing the difference between the achievement of minorities and 
nonminorities. 

Of the remaining predictor variables none were statistically significant at the . 10 level; 
however, the average frequency of computer use for the school had a value of £ = . 1 1 . Table 4 
also contains information concerning the reliability, parameter variance, and proportion of 
parameter variance explained. The table shows that the reliability was .12 and the parameter 
variance was still significantly large at 48.66 indicating that there is still some parameter 
variance that can be explained. The table also shows that the model was able to explain 3 1% of 
the original parameter variance. 

The results of the between-schools model for the gender parameters are contained in 
Table 5. The results show that the only predictor that is significantly associated with the gender 
parameter is the average SES level of the student body. The results for this predictor indicate 
that higher levels of SES are associated with larger values of the gender parameter. Like the 
previous interpretation of availability/access of computers and the race/ethnicity parameter, the 
positive coefficient associated with average SES level of the student body indicates thafthe 
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difference between the achievement of females and males is smaller with higher levels of 
average SES. This interpretation can be seen in Figure 7. 



Table 5 

Predictors of gender coefficients for grade 8 



WITHIN SCHOOL PARAMETER 
Between-school predictor 


Coefficient 


T-ratio 


p-value 


GENDER 








Intercept 


- 27.42 


-0.82 


0.409 


Proportion of minority 


-9.94 


-1.39 


0.163 


Proportion of females 


16.70 


0.89 


0.374 


Average SES 


31.43 


1.75 


0.080 


Computer availability/access 


0.35 


0.03 


0.976 


Average frequency of computer use 


3.30 


0.64 


0.523 


Reliability 




.14 




Parameter variance 




44.59* 




Proportion of parameter variance explained 




.00 





Note: * £ s .01 
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The results of Table 5 show that the reliability was .14 and the parameter variance was 
significantly larger than zero again indicating that some parameter variance can still be 
explained. Finally, the results in Table 5 show that the between-schools model used to explain 
the parameter variance surrounding the gender within-school parameter did not explain any of 
the original parameter variance. 

The between-schools model for the within-school SES parameter yielded no significant 
school level predictors. These results and the corresponding HLM statistics are contained in 
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Table 6. The results show that the reliability was . 10 and there is still parameter variance that can 
be explained. Also, as a result of the nonsignificant model, none of the original parameter 
variance was explained. 



Table 6 

Predictors of socioeconomic status coefficients for grade 8 



WITHIN SCHOOL PARAMETER 
Between-school predictor 


Coefficient 


T-ratio 


p-value 


SOCIOECONOMIC STATUS 








Intercept 


82.80 


1.87 


0.061 


Proportion of minority 


-0.58 


-0.06 


0.952 


Proportion of females 


-29.01 


-1.10 


0.272 


Average SES 


-12.85 


-0.54 


0.588 


Computer availability/access 


-3.52 


-0.26 


0.796 


Average frequency of computer use 


-8,67 


-1.19 


0.236 


Reliability 




.10 




Parameter variance 




52.80* 




Proportion of parameter variance explained 




.00 





Note: * £ £ .01 



Interpretations and Implications 

The results of this analysis have far-reaching policy implications for schools in the state 
of Mississippi. In this section, each of the eight questions addressed in this study will be 
answered followed by a discussion of the policy implications for schools. Finally, the 

m, 

conclusions from these findings are presented along with suggestions for further research. 



NAEP-TSA and Computers 26 

The first two questions relate to the relationships between availability/access and use of 
computers and the average school achievement. Question 1 asks if having more computers or 
greater access to computers increases school achievement. The answer, from our analysis, is that 
availability/access to computers is not statistically significantly associated with a change in 
student achievement. However, the impact on student achievement as a result of the frequency of 
computer use can be detected. While statistically speaking the results do not indicate significance 
(p = .18), the evidence leads one to conclude that there is a level of association between the 
frequency of computer use and achievement. While statistically this association is weak, the 
educational implications are important. The conditional model shows that the more frequently 
the computer is used the lower the average math achievement is for a school. This trend is 
shown in Figure 5. 

The analysis of the first two questions also included an examination of the relationships 
between school achievement and student body characteristics, such as the percentage of females, 
minorities, and the average socioeconomic status of the student body. In general, the differences 
in school achievement cannot be explained by the percentage of females or socioeconomic status 
of the student body. However, the proportion of minorities in a school is associated with school 
achievement. As Figure 1 shows, as the proportion of minorities in a school increases the 
average school achievement decreases. 

Questions 3 and 4 focus on the difference between the mathematics achievement of 
minorities and nonminorities. Figure 3 illustrates the association between computer 
availability/access and the minority-nonminority achievement difference. The figure slyaws that 
as computers were made more available and as teachers had greater access to computers the 
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difference between the performance of minorities and nonminorities was narrowed. When one 
looks at the frequency of computer use, it is revealed that in schools where computers were more 
frequently used, there was a wider gap between the average performance of minorities and 
nonminorities. While this gap is not statistically significant (jj = .11), it does show a marked 
trend which is shown in Figure 6. 

The association between the student body characteristics and the difference in minority 
and nonminority achievement were included in the analysis of questions 3 and 4. A significant 
association was found for the average student body SES. Figure 2 indicates that as the SES of a 
student body increases the difference between the achievement levels of minority and 
nonminoritiy students widens. As shown in Figure 2, the difference in the gap approaches zero 
for schools with the lowest average SES. Thus, in Mississippi schools with low SES student 
bodies there is a smaller difference in the performance of minority and nonminority students. 
The race gap analysis also indicated that there was no relationship between the percentage of 
minorities or females in a student body and the difference in the achievement of minorities and 
nonminorities in a school. 

The analysis for questions 5 and 6 revealed no association between the achievement 
difference of female and male students and computer availability/access or frequency of 
computer use. Further, the analysis indicated no relationship between the percentage of females 
and minorities wi thin a school and the gender gap. However, it did indicate an association 
between the average SES level of the student body and the difference between the achievement 
of female and male students. As Figure 4 illustrates, the difference between the achievement of 
female and male students was narrower in schools with higher average socioeconomic status. 
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The final two policy-relevant questions addressed the associations between computer 
availability/access and frequency of computer use and the performance of low and high SES 
students. The analysis revealed no significant associations between the within-school SES 
coefficient and the two policy-relevant computer variables. Additional analyses also indicated 
that the student body characteristics were not significantly associated with the within-school SES 
coefficient. 

In short, these analyses lead to several interesting conclusions and policy implications. 
This research attempted to explain variation in average mathematics achievement for eighth 
grade public school students in Mississippi for the year 1992. While neither availability/access 
nor frequency of computer use was found to be statistically significantly associated with average 
school achievement, both variables showed associations with the gap between minority and 
nonminority students. In fact, in schools where computers were the most available and accessible 
the gap between minority and nonminority students was significantly reduced. 

The implications of this research for state policy makers are provocative. While spending 
more resources to provide more computers for schools may not lead to improved math 
achievement, training teachers on more appropriate uses of existing computers may bring about 
greater improvement in math achievement. For local school policy makers, the implications of 
this research indicate that computer availability/access seems to be related to reducing the gap 
between minority and nonminority students. However, this research does not address the many 
other factors that come into play to improve math achievement. Contrary to some conventional 
wisdom that computers in the schools will improve math achievement, a conclusion of this 




29 



NAEP-TSA and Computers 29 

research is that there is not a strong association between computer availability/access or 
frequency of use and math achievement. 

Recommendation for Further Research 

One of the basic assumptions of Hierarchical Linear Modeling is that relationships are 
linear. As the researchers were studying this model and the characteristics of several variables, it 
was observed that often the assumption of linearity may be violated, and at times, quite severely 
As shown in figures 7, 8, and 9, other nonlinear lines may better fit the data. Specifically in 
Figure 8, the line may decrease between 0 and 0.4, stabilize between 0.4 and 1.0, then decrease 
quite drastically between 1.0 and 1.6. The implications of this violation deserve further 
investigation. 
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